FIG. 11(a) is a diagram for explaining a conventional image decoding apparatus 50.
This image decoding apparatus 50 decodes coded image data corresponding to a single image sequence. The image decoding apparatus 50 includes a decoder 51 and a memory 52. The decoder 51 receives, for example, a video stream based on a standard such as MPEG2 (Moving Picture Experts Group Phase 2) as image data Sb, decodes the image data for each frame, and outputs decoded image data Db. The memory 52 stores the decoded image data of the already decoded frames, as reference image data Dr to be referred to in the decoding process.
In the image decoding apparatus 50, decoding of the inputted coded image data Sb is carried out with reference to the reference image data Dr which is stored in the memory 52, and the decoded image data Db obtained in the decoding process is outputted. The decoded image data Db corresponding to the target frame is stored in the memory 52 as the reference image data Dr for the frames subsequent to the target frame.
In the conventional image decoding apparatus 50 constructed as described above, however, image data which can be decoded simultaneously are limited to those corresponding to a single image sequence. Therefore, the conventional apparatus 50 cannot perform parallel decoding of image data corresponding to plural image sequences, for example, image data corresponding to plural objects (image sequences) which exist in one frame as shown in FIG. 13 (in FIG. 13, four objects Ob0–Ob3).
Thus, there has been proposed a construction of an image decoding apparatus 110 shown in FIG. 11(b), which can perform parallel decoding of image data corresponding to plural image sequences (objects) as shown in FIG. 13.
The image decoding apparatus 110 shown in FIG. 11(b) receives a video stream which includes coded image data Sb0–Sb3 shown in FIG. 14(a) corresponding to the four image sequences Ob0–Ob3 shown in FIG. 13, respectively, for example, a video stream Vs in which the coded image data Sb0–Sb3 are multiplexed as shown in FIG. 14(b), and decodes the coded image data Sb of the respective image sequences in parallel with each other. Although the respective coded image data are multiplexed in regular arrangements in the video stream Vs shown in FIG. 14(b), there are cases, depending on the multiplexing method, where the respective coded image data are randomly arranged according to the frame rates or the like of the respective coded data.
That is, the image decoding apparatus 110 includes: decoders 112a–112d for decoding the respective coded image data Sb0–Sb3 corresponding to the four image sequences, and outputting decoded image data Db0–Db3 corresponding to the respective image sequences; and memories 113a–113b for holding reference image data Dr0–Dr3 which are to be referred to when the respective image sequences are decoded.
Further, the image decoding apparatus 110 includes: an input-side switch 111 for supplying the inputted video stream Vs to one of the four decoders 112a–112d on the basis of image sequence identifying information Id for identifying an image sequence; and an output-side switch 114 for selecting one of the decoded image data Db0–Db3 outputted from the four decoders 112a–112d, and outputting the selected decoded image data Db as reproduced image data EDb.
The input-side switch 111 has an input terminal 111a to which the video stream Vs is applied, and output terminals 111b0–111b3 for outputting the coded image data Sb0–Sb3 to the corresponding decoders 112a–112d. The input-side switch 111 connects the input terminal 111a to one of the output terminals 111b0–111b3 which corresponds to the image sequence indicated by the image sequence identifying information Id. The output-side switch 114 has input terminals 114a0–114a3 to which the decoded image data Db0–Db3 outputted from the respective decoders 112a–112d are applied, and an output terminal 114b for outputting the reproduced image data EDb. The output-side switch 114 connects the output terminal 114b to one of the input terminals 114a0–114a3 which corresponds to the image sequence indicated by the image sequence identifying information Id.
Next, the operation of the conventional image decoding apparatus 110 constructed as described above will be described.
When the video decoding apparatus 110 receives the video stream Vs including the image data Sb0–Sb3 of the four image sequences, and the image sequence identifying information Id for identifying image data in the video stream Vs, the input-side switch 111 connects the input terminal 111a to which the video stream Vs is applied, to a desired output terminal of the input side switch 111 on the basis of the image sequence identifying information Id. For example, when the coded image data Sb0 corresponding to the first image sequence is inputted as the video stream Vs, the image sequence identifying information Id indicates that the video stream Vs is the coded image data Sb0 corresponding to the first image sequence Ob0. Thereby, in the input-side switch 111, the input terminal 111a is connected to the first output terminal 111b0 on the basis of the image sequence identifying information Id, and the coded image data Sb0 is supplied to the first decoder 112a corresponding to the first output terminal 111b0.
In the first decoder 112a, decoding of the coded image data Sb0 is carried out with reference to the reference image data Dr0 that is stored in the first memory 113a, and the decoded image data Db0 obtained in the decoding process is outputted to the first input terminal 114a0 of the output-side switch 114.
Then, in the output-side switch 114, the output terminal 114b is connected to the first input terminal 114a0 on the basis of the image sequence identifying information Id, and the decoded image data Db0 from the first decoder 112a is outputted as the reproduced image data EDb.
When the coded image data Sb1, Sb2, or Sb3 corresponding to the second, third, or fourth image sequence is inputted as the video stream Vs, in the same manner as described above, the input-side switch 111 changes the connecting state between the input terminal and the output terminal thereof according to the image sequence identifying information Id so that the inputted coded image data Sb is supplied to the corresponding decoder 112, and the output-side switch 114 changes the connecting state between the input terminal and the output terminal thereof so that the decoded image data Db from the decoder 112 corresponding to the image sequence indicated by the image sequence identifying information Id is outputted as the reproduced image data EDb. As the result, decoding of the coded image data Sb corresponding to the image sequence indicated by the image sequence identifying information Id is carried out.
In the image decoding apparatus 110 which can decode the plural coded image data Sb in parallel with each other, as shown in FIG. 11(b), the independent decoders 112a–112b are required for the respective image data to be processed in parallel, and a decoder 112 to be used should be selected according to an image sequence which corresponds to the inputted video stream Vs. However, this leads to drawbacks such as a complicated construction of the image decoding apparatus, an increased size of the image decoding apparatus and, further, increased costs due to the complicated and large-sized construction.
Further, in the method of performing parallel decoding on plural image data by using a number decoders as many as the number of the image sequences to be decoded, the number of decoders corresponding to the image sequences to be decoded must be decided in advance.
Furthermore, Japanese Published Patent Application No. Hei.9-093577 discloses an image decoding apparatus which is constructed to perform time division decoding on image data corresponding to plural image sequences by using a single decoder. Hereinafter, the image decoding apparatus disclosed in this literature will be briefly described.
FIG. 12 is a block diagram for explaining this image decoding apparatus.
The image decoding apparatus 200 comprises a decoder 200a and a memory 200b. The decoder 200a performs time division decoding on image data corresponding to plural (four in this case) image sequences, on the basis of a video stream Vs including the image data corresponding to the four image sequences, and image sequence identifying information Id for identifying each image sequence. The memory 200b stores reference image data Dr to be referred to when the decoding is carried out.
The decoder 200a includes dedicated registers 211–214, parameter selectors 221–224, a register selector 230, and a decoding unit 240. The dedicated registers 211–214 correspond to the respective image sequences, and each register stores decoding data which comprises data (parameter data) Dp indicating plural parameters 0–N that are used for decoding. The parameter selectors 221–224 correspond to the respective dedicated registers 211–214, and each parameter selector selects a storage position for each parameter in the corresponding register, on the basis of parameter storage position information Ip indicating the types of the parameters. The register selector 230 selects one of the parameter selectors 221–224 on the basis of the image sequence identifying information Id, thereby selecting one of the dedicated registers 211–214. The decoding unit 240 decodes the video stream Vs, on the basis of the decoding information including the parameter data Dp stored in each register, with reference to the reference image data Dr stored in the memory 200b, and outputs decoded image data Db obtained by the decoding process.
The decoding unit 240 extracts the parameter data Dp (decoding information) corresponding to each frame in each image sequence on the basis of header information included in the video stream Vs, and outputs the parameter data Dp to the register selector 230 and, simultaneously, outputs parameter storage position information Ip indicating the type of the parameter to the respective parameter selectors 221–224.
In the dedicated registers 211–214, the respective parameter data Dp indicating the parameters 0–N are stored in the parameter storage positions R0–Rn corresponding to the respective parameters. Further, the register selector 230 has a first terminal 235 for accessing each dedicated register which stores the decoding information of each image sequence, and plural second terminals 231–234 corresponding to the respective dedicated registers 221–224, for accessing the parameter data Dp stored in the respective registers 211–214. The register selector 230 connects the first terminal 235 to any of the second terminals 231–234 according to the image sequence identifying information Id.
Further, the parameter selector 221 has a first terminal b connected to the second terminal 231 of the register selector 230, and second terminals a0–an for accessing the parameter data Dp stored in the respective parameter storage positions R0–Rn of the register 211. The parameter selector 222 has a first terminal b connected to the second terminal 232 of the register selector 230, and second terminals a0–an for accessing the parameter data Dp stored in the respective parameter storage positions R0–Rn of the register 212. Likewise, the parameter selector 223 has a first terminal b connected to the second terminal 233 of the register selector 230, and second terminals a0–an for accessing the parameter data Dp stored in the respective parameter storage positions R0–Rn of the register 213. The parameter selector 224 has a first terminal b connected to the second terminal 234 of the register selector 230, and second terminals a0–an for accessing the parameter data Dp stored in the respective parameter storage positions R0–Rn of the register 214.
Next, a description will be given of the operation of the image decoding apparatus 200 constructed as described above.
When the video stream Vs and the image sequence identifying information Id are inputted to the image decoding apparatus 200, the video stream Vs is analyzed by the decoding unit 240, whereby the parameter data Dp, as decoding information corresponding to a predetermined image sequence, is extracted from the video stream Vs and outputted to the register selector 230. Further, the parameter storage position information Ip indicating the type of each parameter data Dp which is extracted as decoding information from the video stream Vs is sequentially outputted to the respective parameter selectors 221–224.
In the register selector 230, on the basis of the image sequence identifying information Id, the parameter data Dp from the decoding unit 240 is outputted to a desired parameter selector, for example, the parameter selector 221. In the parameter selector 221, on the basis of the parameter storage position information Ip outputted from the decoding unit 240, the respective parameter data Dp as decoding information from the register selector 230 are sequentially stored in the corresponding parameter storage positions in the exclusive register 211. To be specific, the parameter data Dp corresponding to the parameters 0, 1, . . . , N are stored in the parameter storage positions R0, R1, . . . , Rn in the register 211, respectively.
In the decoding unit 240, on the basis of the parameter data Dp as decoding information stored in the register 211, decoding is performed on the video stream Vs corresponding to a target frame of a predetermined image sequence, with reference to the reference image data Dr stored in the memory 200b. When this decoding is completed, image data corresponding to the already processed frame is stored in the memory 200b as reference image data Dr to be referred to when decoding the following frames.
Further, when the video stream Vs corresponding to another image sequence is inputted, the parameter data Dp as decoding information is stored in the register corresponding to the image sequence and, thereafter, decoding is performed on the video stream Vs corresponding to this image sequence, on the basis of the parameter data Dp stored in the register.
As described above, in the image decoding apparatus 200 shown in FIG. 12, the decoder 200a is provided with the dedicated registers (decoding information storage registers) for storing decoding information, which number as many as the number of image sequences that are to be decoded. Therefore, it is possible to decode the video stream Vs including plural image sequences by using one decoder. However, although the image decoding apparatus 200 can perform parallel decoding of the plural image sequences included in the video stream Vs by using one decoder 200a, the decoder 200a should be provided with the decoding information storage registers which number as many as the number of image sequences that are to be processed.
As described above, in the conventional image decoding apparatus, in order to perform parallel decoding on image data of plural image sequences, it is necessary to prepare decoders which number as many as the number of the image sequences to be processed simultaneously, or it is necessary to provide a decoder having decoding information storage registers which number as many as the number of the image sequences to be processed simultaneously. In other words, in the conventional image decoding apparatus, the number of image sequences to be processed simultaneously must be known in advance.
Accordingly, when the number of image sequences to be processed simultaneously is changed, the construction of the decoder must be reviewed or the decoder must be redesigned according to the change. This causes an increase in the cost of the image decoding apparatus.
Furthermore, when the image decoding apparatus is provided with decoders which number as many as the number of image sequences that are to be processed simultaneously, the construction of the apparatus is complicated, and the size of the apparatus is increased.
Moreover, among MPEG schemes as international standards relating to image data compression techniques, in MPEG4 by which coded image data of plural image sequences are handled simultaneously, coded image data corresponding to plural objects (image sequences) as shown in FIG. 14 are decoded and composited to obtain reproduced data corresponding to a composite image shown in FIG. 13, and the composite image is displayed. Accordingly, in the coding scheme based on MPEG4, image data corresponding to plural objects (image sequences) constituting one scene are multiplexed by packets to be transmitted as a single bit stream. In order to handle such bit stream, parallel decoding of the image data corresponding to the plural image sequences is required.
Accordingly, in the future, as an image decoding apparatus for simultaneously performing parallel decoding processes on image data of plural image sequences, a simple and inexpensive apparatus which can easily cope with a change in the number of image sequences to be handled simultaneously will be increasingly demanded.